1. Field of the Invention
The present invention relates to an image processing method, an image processing apparatus, and a computer readable medium, in which an image processing program is recorded. Specifically, the present invention relates to an image processing method, an image processing apparatus, and a computer readable medium, in which an image processing program is recorded, for discriminating the shapes of predetermined subjects, such as faces, that are included in images.
2. Description of the Related Art
Statistical models are constructed of predetermined subjects, such as human faces and body parts, which are included in images, by employing image data sets that represent the images, in the fields of medical diagnostic images and verification employing physical characteristics. Various methods have been proposed for constructing the statistical models.
An ASM (Active Shape Model), which is capable of representing the positions, shapes, and sizes of each structural element of a predetermined subject, such as cheeks, eyes, and mouths that constitute faces, is disclosed in T. F. Coots, A. Hill, C. J. Taylor, and J. Haslam, “Use of Active Shape Models for Locating Structures in Medical Images”, Image and Vision Computing, pp. 355-365, Volume 12, Issue 6, 1994, and in U.S. Patent Application Publication No. 20040170323. The method for constructing an ASM comprises the following steps. First, as illustrated in FIG. 18, positions of landmarks that represent the positions, shapes, and sizes of each structural element of a predetermined subject (a face, in the example of FIG. 18) are specified within a plurality of sample images of the predetermined subject, to obtain a frame model for each sample image. The frame model is formed by connecting points, which are the landmarks, according to predetermined rules. For example, in the case that the predetermined subject is a face, points along the facial outline, points along the lines of the eyebrows, points along the outlines of the eyes, points at the positions of the irises, points along the lines of the upper and lower lips, and the like are specified as the landmarks. Frames constituted by connecting the points at the positions of the irises, the positions along the facial outline, the points along the lines of the eyebrows and the like comprise the frame model of the face. Frame models, which are obtained from the plurality of sample images, are averaged to obtain an average facial frame model. The positions of each of the landmarks within the average facial frame model are designated to be the average positions of corresponding landmarks of each of the sample images. For example, in the case that 130 landmarks are specified within a face, and the 110th landmark indicates the position of the tip of the chin, the position of the 110th landmark in the average frame model is a position obtained by averaging the positions of the 110th landmark within the sample images. In the ASM technique, the average frame model obtained in this manner is applied to the predetermined subject pictured in a target image, on which image processes are administered. The positions of the landmarks within the applied average frame model are set to be the initial values of the positions of the landmarks within the predetermined subject pictured in the target image. The average frame model is deformed to match the predetermined subject pictured in the target image (that is, the positions of each landmark within the average frame model are moved), thereby obtaining the positions of each landmark within the predetermined subject pictured in the target image. Here, the deformation of the average frame model will be described.
As described above, the frame model that represents the predetermined subject is defined by the positions of the landmarks within the frame model. Therefore, in the case of a two dimensional frame model S, the frame model S can be represented by vectors comprising 2n (wherein n is the number of landmarks) components, defined in Formula (1) below.S=(X1, X2, . . . , Xn, Xn+1, Xn+2, . . . , X2n)  (1)wherein
S: frame model
n: number of landmarks
Xi (1≦i≦n): the X direction coordinate value of an ith landmark
Xn+i (1≦i≦n): the Y direction coordinate value of an ith landmark
The average frame model Sav is defined by Formula (2) below.Sav=( X1, X2, . . . , Xn, Xn+1, Xn+2, . . . , X2n)  (2)wherein
Sav: average frame model
n: number of landmarks
 Xi(1≦i≦n): the average X direction coordinate value of an ith landmark
 Xn+i(1≦i≦n): the average Y direction coordinate value of an ith landmark
The matrix illustrated in Formula (3) below can be derived, employing frame models of each sample image and the average frame model Sav obtained therefrom.
                    [                                                                              ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    2                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                  ⋯                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                                                  2                          ⁢                          n                                                j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                                                                                                                            ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                  ⋯                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                                      (                                                                  X                                                  2                          ⁢                          n                                                j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                                                                                          ⋮                                      ⋮                                      ⋮                                      ⋮                                                                                            ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                    ⁢                  ⋯                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                    2                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                    ⁢                                      (                                                                  X                                                  2                          ⁢                          n                                                j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                                                                                                                            ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        1                        j                                            -                                                                        X                          _                                                1                                                              )                                    ⁢                                      (                                                                  X                                                  2                          ⁢                          n                                                j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                2                                                              )                                    ⁢                  ⋯                                                                                                      ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                                                                              2                            ⁢                            n                                                    -                          1                                                j                                            -                                                                        X                          _                                                                                                      2                            ⁢                            n                                                    -                          1                                                                                      )                                    ⁢                                      (                                                                  X                                                  2                          ⁢                          n                                                j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                                                                                                        ∑                                      j                    =                    1                                    m                                ⁢                                                      (                                                                  X                        2                        j                                            -                                                                        X                          _                                                                          2                          ⁢                          n                                                                                      )                                    2                                                                    ]                            (        3        )            wherein
n: number of landmarks
m: number of sample images
Xij(1≦i≦n): the X coordinate value of an ith landmark within a jth sample image
Xn+ij(1≦i≦n): the Y coordinate value of an ith landmark within a jth sample image
 Xi(1≦i≦n): the average X coordinate value of an ith landmark
 Xn+i(1≦i≦n): the average Y direction coordinate value of an ith landmark
K (1≦K≦2n) unique vectors Pj (Pj1, Pj2 . . . , Pj(2n)) (1≦j≦K), and unique values λj (1≦j≦K), corresponding to each of the unique vectors Pj, are derived from the matrix illustrated in Formula (3). Deformation of the average frame model Sav is performed employing the unique vectors Pj, according to Formula (4) below.
                                          S            h                    =                      Sav            +                          Δ              ⁢                                                          ⁢              S                                      ⁢                                  ⁢                              Δ            ⁢                                                  ⁢            S                    =                                    ∑                              j                =                1                            K                        ⁢                                          b                j                            ⁢                              P                j                                                                        (        4        )            wherein
Sh: the frame model after deformation
Sav: the average frame model
ΔS: the amount of positional movement of a landmark
K: number of unique vectors
Pj: a unique vector
bj: a deformation parameter
ΔS in Formula (4) represents the amount of positional movement of each landmark. That is, the deformation of the average frame model Sav is performed by moving the positions of each landmark. As can be understood from Formula (4), the amount of positional movement ΔS of each landmark is derived from a deformation parameter bj and a unique vector Pj. The unique vector Pj is obtained in advance. Therefore, it is necessary to obtain the deformation parameter bj, in order to deform the average frame model Sav. Here, obtainment of the deformation parameter bj will be described.
The deformation parameter bj is obtained in the following manner. First, characteristic amounts that specify each of the landmarks are obtained for each landmark within each of the sample images. Here, an example will be described in which the characteristic amount is a brightness profile, and the landmark is that which is positioned at the recess of an upper lip. The landmark positioned at the recess of the upper lip is the center of the upper lip (point A0 in FIG. 19A). A first line that connects the landmarks adjacent thereto at both sides thereof (points A1 and A2 in FIG. 19A) is drawn. A line L perpendicular to the first line and that passes through the landmark A0 is drawn. The brightness profile within a small region (for example, 11 pixels) along the line L with the landmark A0 at its center is obtained as the characteristic amount of the landmark A0. FIG. 19B is a graph that represents the brightness profile, which is the characteristic amount of the landmark A0 of FIG. 19A.
A totalized characteristic amount that specifies landmarks positioned at the recesses of upper lips is obtained from the brightness profiles of the landmarks positioned at the recesses of upper lips within each of the sample images. There are differences among the characteristic amounts of corresponding landmarks (for example, landmarks positioned at the recesses of upper lips) within the plurality of sample images. However, the totalized characteristic amount is obtained on the assumption that a Gaussian distribution is present for the characteristic amounts. An averaging process is an example of a method for obtaining the totalized characteristic amount based on the assumption that a Gaussian distribution is present. That is, the brightness profiles of each landmark within each of the plurality of sample images are obtained. The brightness profiles of corresponding landmarks among the plurality of sample images are averaged, and designated as totalized characteristic amounts. That is, the totalized characteristic amount of the landmark positioned at the recess of an upper lip is obtained by averaging the brightness profiles of landmarks positioned at the recesses of the upper lips within the plurality of sample images.
Deformation of the average frame model Sav to match the predetermined subject, which is included in the target image, in the ASM technique is performed in the following manner. First, a point which has a characteristic amount most similar to the totalized characteristic amount of a corresponding landmark is detected from within a region of the target image that includes the corresponding landmark of the average frame model Sav. For example, in the case of the recess of an upper lip, a region, which is greater than the aforementioned small region, that includes the position corresponding to the landmark at the recess of the upper lip in the average frame model Sav is set within the target image. The region may be, for example, 21 pixels along the aforementioned line L, having the first position at its center. Brightness profiles are obtained from within 11 pixel regions having each of the pixels within the 21 pixel region as their centers. A brightness profile which is most similar to the totalized characteristic amount (that is, the average brightness profile) is selected from among the obtained brightness profiles. The distance between the position at which this brightness profile was obtained (the center pixel of the 11 pixels for which the brightness profile was obtained) and the first position is obtained. The amount of positional movement, for the position of the landmark at the position of the recess of the upper lip within the average frame model Sav to be moved, is determined based on the distance. The deformation parameter bj is calculated from the amount of positional movement. Specifically, the amount of positional movement is determined to be less than the distance, for example, ½ the distance. The deformation parameter bj is calculated from the amount of positional movement.
Note that the amount of positional movement of the landmarks is limited by employing unique values λj, as indicated in Formula (5) below. The amount of positional movement is limited so that the frame model obtained after deforming the average frame model Sav will still be able to represent a face.3√{square root over (λj)}≦bj≦3√{square root over (λj)}  (5)wherein
bj: deformation parameter
λj: unique value
In this manner, the ASM technique deforms the average frame model Sav by moving the position of each landmark, until the deformed frame model converges onto the predetermined subject pictured in the target image. Thereby, a frame model of the predetermined subject pictured in the target image, defined by the positions of each landmark after deformation, is obtained.
However, the aforementioned ASM technique obtains the totalized characteristic amounts from corresponding landmarks within the plurality of sample images, based on the assumption that Gaussian distributions apply to the characteristic amounts of corresponding landmarks. Therefore, the ASM technique cannot be applied to cases in which Gaussian distributions do not apply, such as when variation in characteristic amounts of corresponding landmarks is great, or when illumination conditions differ among sample images. For example, brightness profiles of landmarks positioned at the recess of the upper lip differ greatly depending on whether the subject has a moustache or not. In these cases, a Gaussian distribution does not apply. For this reason, if average brightness profiles, for example, are obtained as totalized characteristic amounts, based on the assumption that Gaussian distributions apply, and landmarks within the target image are detected employing these totalized characteristic amounts, the detection accuracy is poor, and the process is not robust.